Providing energy-efficient features using human presence detection

ABSTRACT

Disclosed herein are system, method, and computer program product embodiments for the detection of human presence in an energy efficient manner using a plurality of sensors such as those of a battery-powered device such as a television remote, and a device with a processor, such as a television. Data gathered from an initial television WiFi radio scan, or an initial low-powered detection scan from the television remote, may be analyzed by the processor to determine a potential presence of one or more humans are present proximate to the device. If there is such a potential presence, the device remote can enter a full-powered detection mode to accurately determine the presence or absence of one or more humans, and take further actions.

RELATED APPLICATIONS

This application is a continuation and claims benefit of U.S. patentapplication Ser. No. 15/930,712 filed May 13, 2020, the content of whichis herein incorporated by reference in entirety.

BACKGROUND

In the present day and age, consumers of entertainment often utilizetelevision-speaker setups. In particular, to achieve a surround-soundeffect, users often arrange a viewing space with speakers, which may beconnected to a central television (TV) at the front of said viewingspace. In this manner, by having sound emanate from various pointsaround them, consumers are able to enjoy entertainment with realisticsound effects.

However, their experience with such a viewing experience may still belimited in several ways. Because it is not known whether humans arepresent or not in front of the TV, the setup cannot often be used forfurther customizations with regard to human presence. For example, thespeakers are not rearranged to match the location where any humans arepresent for optimal sound, lighting is often not arranged to reflect thelocation of the user, etc. In this manner, the user does not receive atailor-made experience that may reflect their position, or theirpersonal liking.

Further, without knowing if a human is present or not in front of thetelevision, additional information such as the number of such humans, orhow they would respond to content playing on the screen, cannot bedetermined. Further, safety and energy features based on such presencedetection can also not be used. Energy is often wasted when a user turnson a TV and may have to leave the front of the TV for a substantialperiod of time, and other peripheral battery-powered devices such as adevice remote for controlling the television are also left on operatingat full-power. Often, perhaps because of an emergency, or due to anyother reason where the user is not able to turn the TV off before theywalk away, a significant amount of energy is wasted in this manner. Inthis case, until the user physically returns to within proximate rangeof the TV, or uses the remote of such a TV, the TV is not turned off. Inthis case, even though the television is not being watched by the user,energy is being wasted.

Furthermore, energy waste may occur regardless of whether the user ispresent or not when a device, such as a WiFi remote for controlling thetelevision, is operating at a full power mode all the time.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are incorporated herein and form a part of thespecification.

FIG. 1 is a block diagram of the interaction between an apparatus moduleand a central module, according to some embodiments.

FIG. 2 is a reference setup of a plurality of interactivesensors/devices forming a zone around a potential user space, in frontof a television, according to some embodiments.

FIG. 3 is a reference setup showing virtual transmitter capabilities ofeach of the sensors/devices, and outflows as a result of transmissionsfrom a virtual transmitter, according to some embodiments.

FIG. 4 shows a presence detection mode of a reference setup, wherein thetransmission of sensors in the setup occurs with humans present in frontof a television, according to some embodiments.

FIG. 5 shows a presence detection mode of a reference setup in apeer-to-peer communication mode between the present sensors/devices,wherein a human may be present in front of a television, according tosome embodiments.

FIG. 6A is a layer-by-layer flow diagram of an exemplary deep learningneural network model with back-propagation that may be used forperforming human presence detection based on stored data from thesensors/devices of a reference setup, according to some embodiments.

FIG. 6B is a flow diagram of an exemplary machine learning modelutilizing multiple classifiers to form a composite combined classifierfor performing human presence detection based on stored data from thesensors/devices of a reference setup, according to some embodiments.

FIG. 7A is an exemplary graphical user interface (GUI) displayed,seeking user input through the GUI to determine if the user is presentproximate to the sensors/devices of a reference setup, according to someembodiments.

FIG. 7B is an exemplary GUI informing a user that an intruder has beendetected, and seeking input from the user on what action to perform withrespect to the intruder, according to some embodiments.

FIG. 8 is an exemplary embodiment of an WiFi and infrared remotecontrol, with buttons enabling a user of the television to provide userinput to the television, according to some embodiments

FIG. 9 is a flow chart illustrating a process for an embodiment whereinput is gathered from numerous sensors/devices in a reference setup infront of a television, and then performing human presence detection inone of two modes, according to some embodiments.

FIG. 10 is a flow chart illustrating a process for an embodiment foractions that may be taken by the sensors/devices or television based onthe presence or absence of a user in front of the television, accordingto some embodiments.

FIG. 11 is a block diagram of an example where a cloud computingenvironment may be accessed by an apparatus module, according to someembodiments.

FIG. 12 is an example computer system useful for implementing variousembodiments.

FIG. 13 is a reference setup of a plurality of where a device WiFiremote may be used in conjunction with interactive sensors/devices toform a zone around a potential user space, in front of a television,according to some embodiments.

FIG. 14 is a flowchart illustrating actions that are taken by the deviceremote based on the television being turned on or off to detect thepresence or absence of a user in front of the television, according tosome embodiments.

In the drawings, like reference numbers generally indicate identical orsimilar elements. Additionally, generally, the left-most digit(s) of areference number identifies the drawing in which the reference numberfirst appears.

DETAILED DESCRIPTION

Provided herein are system, apparatus, device, method and/or computerprogram product embodiments, and/or combinations and sub-combinationsthereof, for efficient human-presence detection or monitoring by Wi-Firadiation using transmitters and receivers as part of a smart,Internet-of-Things (IOT) system, including in speaker-based ortelevision-speaker-based setups. By having multiple speakers or sensorsplaced strategically surrounding a user, the embodiments provided hereincan accurately detect whether or not a user is in front of and proximateto the TV with a great degree of accuracy due to multiple signals beingsent and is versatile for use in a variety of locations. Further, basedon such detection, and embodiments where such speakers are smartspeakers and can interact with a smart television, a variety ofuser-tailored experiences can be provided such as home automation,detection of a user coming home, environmental features such as a greenmode or a sleep mode, and the detection of an intruder.

FIG. 1 is a block diagram of a data-transfer environment 100 showing theinteraction between an apparatus module 102, with a central television(TV) module (central module) 104, which may include a central modulerepository 104 a. The embodiments described herein are directed toapparatus modules including a plurality of speakers and other devicessuch as smartphones. WiFi remotes, soundbars, etc. and a television, ina television surround sound set-up, where these devices collectively mayform a smart, Internet-of-Things system. In this sense, each of thedevices has sensors attached to it and can transmit data wirelessly toeach other, or to a television, which in this case is the central module104. Through the described embodiments, smart speakers may be able,through WiFi transmissions and receptions from each speaker, incombination with data from sensors in a user-based cell phone,user-based remote input, etc., to transmit data to the central modulefor processing, and determine whether one or more humans are present infront of a TV. Alternately, each of the devices may submit data to thecentral module 104, which may be sent as part of a cloud-based networkto a backend for further processing, or if each of the devices isconnected to the internet, may be submitted from the device to thecloud-based network directly, wherein a result may then be submittedback to the central module 104. The outcome may drive events andcustomizations undertaken by the central module 104, utilizing apparatusmodules 102.

According to an embodiment, the central module 104 and the apparatusmodule 102 may comprise one or more separate computer systems such asthe computer system 1200, as shown in FIG. 12 and will be explainedlater.

To aid in describing the methods which are explained in FIGS. 2-10 thatfollow, an example embodiment of the underlying structure will first bedescribed. The underlying structure of a computer system 1200, shown inFIG. 12 , can be implemented by a smart television 202 as shown in FIG.2 . Such a computer system, may, according to the embodiments describeabove, include apparatus modules 102 as sensor-based subcomponents,central module 104, and central module repository 104 a. Computer system1200 may include one or more processors (also called central processingunits, or CPUs), such as a processor 1204, which may be a processorbuilt-in the television (e.g. 202 in FIG. 2 ). Processor 1204 may beconnected to a communication infrastructure or bus 1206.

Computer system 1200 may be virtualized, such as a virtual machine on alocal server, or as part of a cloud-based network as described above. Onthe other hand, instead of being virtualized, in an alternativeembodiment it may also include user input/output devices 1203, such as aWiFi remote 206, an application on a user smartphone 204, the screen ofthe television 202 as an output display device, etc., which maycommunicate with communication infrastructure 1206 through userinput/output interface(s) 1202.

One or more processors 1204 may be a graphics processing unit (GPU). Inan embodiment, a GPU may be a processor that is a specialized electroniccircuit designed to process the multitude of data received from numerousspeakers 210 a-210 f, device remote 206, and a user phone 204 in thereference setup 200 shown in FIG. 2 . This data may be processed by theGPU utilizing a threshold-based, logistic regression, ormachine-learning model to determine whether or not one or more humansare in front of the TV. Further, once such a determination is made, sucha GPU may also be used for multiple user-based customizations. Suchcustomizations may include controlling the position and output ofspeakers 210 a-210 f for optimal sound relative to a user location,performing an ambient sunrise on the screen of a TV (see 402 a in thepresence detection setup 400 of FIG. 4 ), controlling the position,output, and hue of lights 412 a and 412 b, displaying graphic userinterfaces (GUI)'s on the television screen of TV 202 or on the app of auser phone 208 as shown in FIG. 7 , entering a full-power or low-powermode on battery powered devices such as the device remote 206 if theuser is asleep or not, or processing continuously monitored device andsensor data for detecting the presence of a possible intruder.

The GPU may have a parallel structure that is efficient for parallelprocessing of large blocks of data, such as data received from speakers,the device remote, or from an application on a user's phone, orconversely, for displaying graphics on the television screen of thesmart television 202 or controlling sound or light output, etc.Mathematically intensive data is common to such computer graphicsapplications, images, videos, and audio output any of which can includetable data received and stored in central module repository 104 a.

A smart television implementing the computer system 1200 can alsoinclude a memory for use in connection with the execution of applicationprogramming, for receiving input from the devices such as wirelessspeakers 210 a-210 f, WiFi remote 206 or user phone 204, of FIG. 2 , andfor the temporary or long term storage of program instructions and/ordata. As examples, main or primary memory 1208 of the TV can include,such as random access memory (RAM), dynamic random access memory (DRAM),synchronous dynamic access memory (SDRAM), or other solid state memory.Main memory 1208 can include one or more levels of cache (includingsecondary cache), which may be used in applications which controlspeaker output, light output, or output graphics on the display screenof a smart television such as 202 of FIG. 2 , as well as for providingenergy efficient features and modes of operation.

Computer system 1200 can also include one or more secondary storagedevices or memory 1210. Secondary memory 1210 may include, for example,an external hard disk drive 1212 and/or a removable storage device ordrive 1214 which can be plugged into the TV, which may interact with aRaid array 1216, which may combine multiple physical hard disk drivecomponents (such as SSD or SATA-based disk drives) into one or morelogical units, or a removable storage unit 1218. Removable storage unit1218 may include a computer usable or readable storage device havingstored thereon computer software (control logic) and/or data, includingremotely accessed network drives. Removable storage unit 1218 may alsobe a program cartridge and cartridge interface, a removable memory chip(such as EPROM or PROM) and associated socket, a memory stick and USBport, a memory card and associate memory card slot, and/or any otherremovable storage unit and associated interface. Removable storage drive1214 may read from and/or write to removable storage unit 1218.Alternatively, or in addition, secondary memory 1210 may also benon-removable and built into the TV, and comprise a solid state memorydevice or other devices.

A TV implementing computer system 1200 may further include acommunication or network interface 1224. Communication interface 1224may enable the TV implementing computer system 1200 to communicate andinteract with any combination of external devices, external networks,external entities, etc. (individually and collectively referenced byreference number 1228). In particular the communication interface 1224of such a TV may contain a wireless network interface controller fortransmitting and receiving data using a variety of protocols such asWi-Fi, Bluetooth, etc. In this manner, communication interface 1224 mayallow the TV implementing computer system 1200 to communicate withexternal or remote entities 1228 over communications path 1226, whichmay be wired and/or wireless (or a combination thereof), and which mayinclude any combination of LANs, WANs, the Internet, etc. Control logicand/or data may be transmitted to and from the TV implementing computersystem 1200 via communication path 1226.

The TV implementing the computer system 1200 may also be part of anInternet-of-Things system involving a plurality of apparatus modules102, as described above, for components such as speakers 210 a-210 f,and/or the Wi-Fi remote 206, and/or the user phone 204. The apparatusmodules 102 may comprise computer systems 1200 in their own right, wherefor example they are each at least equipped with their own wirelessnetwork interface microcontroller and RAM, and can receive and transmitsignals using Wi-Fi or Bluetooth protocol. Alternatively, such modules102 may only comprise sensors and transmit/receive data without thepresence of RAM and/or their own wireless network interfacemicrocontroller. In this case, any signal processing may be conductedsolely onboard the TV central module 104. For the embodiments describedherein, the former configuration, with apparatus modules 102 eachcomprising computer systems 1200 in their own right, is used.

Any applicable output data structures, file formats, and schemas incomputer system 1200 may be derived from standards including but notlimited to JavaScript Object Notation (JSON), Extensible Markup Language(XML), Yet Another Markup Language (YAML), Extensible Hypertext MarkupLanguage (XHTML), Wireless Markup Language (WML). MessagePack, XML UserInterface Language (XUL), Structured Query Language (SQL), or any otherfunctionally similar representations alone or in combination, and may beused for sending or receiving data (e.g. between any of the apparatusmodules 102, the central module 104, and the central repository 104 a inFIG. 1 ). Alternatively, proprietary data structures, formats or schemasmay be used, either exclusively or in combination with known or openstandards.

In some embodiments, a tangible, non-transitory apparatus or article ofmanufacture comprising a tangible, non-transitory computer useable orreadable medium having control logic (software) stored thereon may alsobe referred to herein as a computer program product or program storagedevice. This includes, but is not limited to, computer system 1200, mainmemory 1208, secondary memory 1210, and removable storage units 1218 and1222, as well as tangible articles of manufacture embodying anycombination of the foregoing. Such control logic, when executed by oneor more data processing devices (such as computer system 1200), maycause such data processing devices to operate as described herein.

Computer system 1200 may be a client or server, accessing or hosting anyapplications and/or data through any delivery paradigm, including butnot limited to remote or distributed cloud computing solutions such ascloud computing environment 1101 which will be explained infra; local oron-premises software (“on-premise” cloud-based solutions): “as aservice” models (e.g., content as a service (CaaS), digital content as aservice (DCaaS), software as a service (SaaS), managed software as aservice (MSaaS), platform as a service (PaaS), desktop as a service(DaaS), framework as a service (FaaS), backend as a service (BaaS),mobile backend as a service (MBaaS), infrastructure as a service (IaaS),etc.); and/or a hybrid model including any combination of the foregoingexamples or other services or delivery paradigms.

In implementing the central module repository 104 a, as an exampleapproach, for storing and accessing its constituent data objects, the TVimplementing the computer system 1200 may use an in-memory database withpersistence, which may store and access data objects from the primarymemory 1208 of the computer system 1200 with a transaction log forpersistence being stored in secondary memory 1210. Alternatively, the TVimplementing the computer system 1200 may implement only part of thedata as an in-memory database, using less primary memory 1208 than asdescribed above, to reduce the in-memory footprint, and may insteadstore a larger portion of the data as a disk-based database within thesecondary memory 1210, where the data may thus be stored in a tieredmanner (more frequently accessed data may be stored in primary memory1208 while less frequently accessed data may be stored in secondarymemory 1210, for example).

Alternately, the TV implementing the computer system 1200 may store noneof the data in primary memory 1208 or secondary memory 1210, and thecomputer system 1200 may write data to a separate file archive stored ina cloud network, or in the secondary memory (e.g., in a file on a harddrive in a Raid array 1216, on a USB drive 1218, or other type ofsecondary memory 1210, etc.).

Further, if the apparatus modules 102 are implemented as a separatesystem 1200, it may utilize a separate wireless network interfacemicrocontroller with processor 1204 and main memory 1208, as well ason-board secondary memory 1210 which may be optionally present, to senddata through a wireless network interface 1224, wherein the apparatusmodules 102 and central module 104 may comprise entities 1228 present onan internal or external network, collectively forming anInternet-of-Things system, which may be accessed through communicationspath 1226. Similar sending of data may occur in an analogous manner withregards to the central module repository 104 a, and the central module104.

As shown in FIG. 11 , cloud computing environment 1101 may containbackend platform 1104, in a block diagram of an example environment 1100in which systems and/or methods described herein may be implemented. Thecentral module 104 of FIG. 1 , described above, may also include a hostsuch as cloud computing environment 1101. The cloud computingenvironment 1101 may be accessed by the TV central module computingsystem 1102, implementing the same type of computing system 1200 asdescribed above. In this case, the central module computing system 1102of FIG. 11 may access the cloud computing environment 1101 by acommunication or network interface 1124 as shown in FIG. 11 , wherein anetwork gateway 1103 may comprise a remote entity 1228 accessed by thecommunications path 1226 of the central module computing system (wherethe three entities 1101, 1102, and 1103 shown in FIG. 11 wouldcollectively correspond to the central module 104 of FIG. 1 ).Alternately, the computing cloud environment 1101 itself may correspondto a remote entity 1228 in FIG. 12 , and may be accessed directly by thecentral module computing system 1102 through a communications path 626,for example through an application protocol interface (API), eliminatingthe need for a network gateway 1103 (both options are shown in FIG. 11 ,wherein the flow path above the central module computing system 1102uses a network gateway 1103, and the flow path below the central modulecomputing system 1102 connects directly to the cloud computingenvironment 1101, both shown using dashed bi-directional lines). Inaddition, as an alternative to wherein data may be transmittedcollectively from the central module repository 104 a, of central module104 by the corresponding TV module 1102, data may be transmitted to thecloud computing environment 1101 from apparatus modules 102 individuallyusing their respective wireless network interfaces 1224, in a directmanner sing the API described above.

The devices of the environments 1200 and 100 may be connected throughwired connections, wireless connections, or a combination of wired andwireless connections, although as described hereafter a wirelessconnections embodiment is used.

In an example embodiment, one or more portions of the data transferenvironment 100 may be an ad hoc network, an intranet, an extranet, avirtual private network (VPN), a local area network (LAN), a wirelessLAN (WLAN), a wide area network (WAN), a wireless wide area network(WWAN), a metropolitan area network (MAN), a portion of the Internet, aportion of the Public Switched Telephone Network (PSTN), a cellulartelephone network, a wireless network, a WiFi network, a WiMax network,any other type of network, or a combination of two or more suchnetworks.

As explained above, the central module 104 of FIG. 1 may have a centralmodule computing system 1102 as shown in FIG. 11 as TV module 1102 whichcomprises and implements a computer system of the same type as thecomputer system 1200 as shown in FIG. 12 . The apparatus module 102 mayaccess the central module 104 through the central module computingsystem 1102, wherein the apparatus module 102 may be external networkentities 1228 from the perspective of the central module computingsystem 1102 in an embodiment, and may send data back and forth in theform of data packets through the communications path 1226 of thecommunications interface 1224 of system 1102, using e.g.,TCP/UDP/FTP/HTTPS/HTTP protocol. If a wireless network interface isused, e.g., then transmission of such data may be in conformance withIEEE 802.11 standards for WLAN transmission, or alternatively may useBLUETOOTH protocol. The network interface 1103, may also be a similarcommunications interface as 1224 with a communications path 1226 forcommunication of data between the TV module 1102 and the backendplatform 1104 of a cloud computing environment 1101 comprising computingresources 1105. Such a backend may be a local server, or hostedremotely, and may also be accessed using e.g., TCP/UDP/FTP/HTTPS/HTTPprotocol, WLAN protocol, or Bluetooth protocol, etc.

Alternately or in addition, the apparatus module 102, such as a userphone 204 shown in FIG. 2 may access the central module 104 through afront-end web-application application 1105 a (e.g. a web browserapplication, a web browser extension, proprietary OS application,standalone executable application, command line access shell program,FTP/UDP/TCP/HTTPS/HTTP protocol, etc.) hosted as an application 1105 aon a computing resource 1105 (explained infra) within the cloudcomputing environment 1101 hosted by the central module 104, in anembodiment. The same application is shown corresponding to element 208in FIG. 2 . Such an application may comprise a software as a service(SaaS) type offering, as described above.

The backend platform 1104 in FIG. 5 may include a server or a group ofservers. In an embodiment, the backend platform 1104 may host a cloudcomputing environment 1101. It may be appreciated that the backendplatform 1104 may not be cloud-based, or may be partially cloud-based.

The cloud computing environment 1101 includes an environment thatdelivers computing as a service (“CaaS” as described above), wherebyshared resources, services, etc. may be provided to the TV module 1102and/or the backend platform 1104. This is especially useful in thecontext of a television, as it adds processing power and additionalphysical resources for the execution of machine learning algorithms andgeneration of output in the form of computer graphics or sound/lightoutput or phone application data, as will be described. The cloudcomputing environment 1101 may provide computation, software, dataaccess, storage, and/or other services that do not require end-userknowledge of a physical location and configuration of a system and/or adevice that delivers the services. For example, the central modulecomputing system 1102, as well as apparatus module 102, may receive datastored within or hosted on a database within computing resources 1105within the backend platform 1104, through an application protocolinterface (API) or any of the various communication protocols previouslylisted. The cloud computing environment 1101 may include computingresources 1105.

Each computing resource 1105 includes one or more personal computers,workstations, computers, server devices, or other types of computationand/or communication devices of the type such as computer system 1200described above. The computing resource(s) 1105 may host the backendplatform 1104. The cloud computing resources may include computeinstances executing in the cloud computing resources 1105. The cloudcomputing resources 1105 may communicate with other cloud computingresources 1105 via wired connections, wireless connections, or acombination of wired or wireless connections.

Computing resources 1105 may include a group of cloud resources, such asone or more applications (“APPs”) 1105 a, one or more virtual machines(“VMs”) 1105 b, virtualized storage (“VS”) 1105 c, and one or morehypervisors (“HYPs”) 1105 d.

An application 1105 a may include one or more software applications thatmay be provided to or accessed by a computer system 1200. In anembodiment, the central module 104 may only include a cloud computingenvironment 1101 executing locally on a TV implementing a computersystem 1200 in the form of the central module computing system 1102. Theapplication 1105 a may include software associated with backend platform1104 and/or any other software configured to be provided across thecloud computing environment 1101 (e.g. to an apparatus module 102). Theapplication 1105 a may send/receive information from one or more otherapplications 1105 a, via one or more of the virtual machines 1105 b.Computing resources 1105 may be able to access each other's applications1105 a through virtual machines 1105 b, in this manner.

Virtual machine 1105 b may include a software implementation of amachine (e.g., a computer) that executes programs like a physicalmachine. Virtual machine 1105 b may be either a system virtual machineor a process virtual machine. A system virtual machine may provide acomplete system platform that supports execution of a complete operatingsystem (OS). A process virtual machine may execute a single program andmay support a single process. The virtual machine 1105 b may execute onbehalf of a user (e.g., the administrator of the central module 104)and/or on behalf of one or more other backend platforms 1104, and maymanage infrastructure of cloud computing environment 1101, such as datamanagement, synchronization, or long duration data transfers.

Virtualized storage 1105 c may include one or more storage systemsand/or one or more devices that use virtualization techniques within thestorage systems or devices of computing resource 1105. With respect to astorage system, types of virtualizations may include blockvirtualization and file virtualization. Block virtualization may referto abstraction (or separation) of logical storage from physical storageso that the storage system may be accessed without regard to physicalstorage or heterogeneous structure. The separation may permitadministrators of the central module 104 flexibility in how they managestorage for evaluation data for processing of data retrieved fromapparatus modules 102 (as will be explained infra). File virtualizationmay eliminate dependencies between data accessed at a file level andlocation where files are physically stored. This manner of block andfile virtualization may enable optimization of storage use, serverconsolidation, and/or performance of non-disruptive file migrations.

Hypervisor 1105 d may provide hardware virtualization techniques thatallow multiple operations systems (e.g., “guest operating systems”) toexecute concurrently on a host computer, such as computing resource1105, which may include a computing system of the type of computingsystem 1200, and can in this manner host a virtualized hardware of acentral module computing system 1102. Hypervisor 1105 d may present avirtual operating platform to the guest operating systems, and maymanage multiple instances of a variety of operating systems as these“guest operating systems,” which may share virtualized hardwareresource, such as RAM, which may for instance access the data in theform of a database of the central module repository (104 a in FIG. 1 ).Alternately, secondary memory may be accessed using virtualized storage1105 c, or on physical storage, such as the hard disk drive 1212, of acomputing resource 1105 of the type of computing system as computingsystem 1200. In embodiments heretofore described, using a combination ofRAM and secondary memory to access the database, such that a portion ofthe database may be in-memory and a portion of the database stored infiles, is also envisioned.

FIGS. 2 through 5 show different facets of how an embodiment of anexemplary reference set up may be used for presence detection. Withreference to FIG. 2 , as described above, television 202 may correspondto a central module 104, and the other components such as speakers 210a-210 f, WiFi remote 206, and a user phone 204 may each correspond to aseparate apparatus module 102 as shown in FIG. 1 . As described above,each of the apparatus modules 102 may have its own wireless networkinterface and accompanying microcontroller, wherein the wireless networkinterface may include, e.g., a USB interface, or an interface card orchip, and an internal antenna. In an embodiment, such an antenna may bean omnidirectional antenna.

In this manner, as shown in FIG. 3 , each of these apparatus modules 102may form a virtual transmitter in its own right, wherein the internalantenna of each device can be used to broadcast a signal in multipledirections. In the embodiment using the omnidirectional antenna, asshown in FIG. 3 , such a signal can be broadcast in all directions.Additionally, the presence of Radio Frequency (wherein WiFi andBLUETOOTH signals are transmitted within the Radio Frequency range)reflective surfaces such as glass, etc., only serves to enhance thesignal further by reflecting it throughout e.g. a room where the setupis used.

In an embodiment, the network interface 1224 may support transmission inaccordance with IEEE 802.11 protocols using the 2.4 ghz or 5 ghzfrequencies, extending to a range of about 150 feet, and 50 feet,respectively. Each virtual transmitter may form a virtual access point,wherein signal transmission is then occurring at multiple pointsthroughout the room in the reference set-up as shown in FIG. 2 . In thiscase, because of the numerous access points present within the perimeterof zone 212 as shown in FIG. 2 , a high-resolution detection zone iscreated, where the transmission of multiple wireless signals allows foreasy triangulation of signal in such a zone. In particular, for example,such signals broadcasted from each of the apparatus modules 102 may bereceived by the network interface 1224 of all other apparatus modules102, or the television 202, in the form of received signal strengthindicator (RSSI). Such an indicator can be translated by the networkinterface controller into an absolute measurement of decibels relativeto a milliwatt (dbM).

Speakers 210 a-210 d, as well as WiFi remote 206, user phone 204, etc.,may be displaced at different three-dimensional locations relative toeach other. In an embodiment, speakers 210 a-210 d may be in a plane atthe same height. In this manner, if for example four of these speakersact as virtual transmitters and broadcast wireless signals in anomnidirectional manner, a wireless network interface receiving thesesignals (e.g. the interface of the television 202) can be used to recordthese dbM measurements and store them in the central module repository104 a. From these stored dbM measurements, a composite map of signalstrength could be created and stored in the central module repository104 a. Especially where the physical distance between television 202 andspeakers 210 a-210 d is known, the corresponding direction strength ofthe wireless signals can be correlated with the respective locationsfrom which they were sent, and provide a strong mapping detection for aparticular height or Z-direction-coordinate, since the multiple sensormeasurements are corroborated with each other at the same height.

Alternatively, other devices such as the remote 206 which may have anaccelerometer and gyroscope may also be used in a similar manner, orapparatus modules 102 such as speakers 210 a-210 d may also have GPSsensors or other positioning sensors which can output a sending positionto a receiving wireless network interface 1224, such as that oftelevision 202. Then, the television 202, which may also have acorresponding GPS, can determine the corresponding distance between thetransmitting speaker or other transmitting device and the television,and can correlate this distance in three dimensions with the dBM datareading. In this manner, a map in three-dimensions can be made of signalstrength. Also or alternatively, a user cell phone 204 may use in-builtsensors such as an accelerometer or gyroscope to determine the relativeposition of the phone. Further, such a phone 204 may receive signalsfrom a multitude of wireless transmitting points (e.g. speakers 210a-210 f) and may through interpolation of its position and receivedsignal strength be able to approximate the distance between eachtransmitting point and the phone, and thus be able to estimate theposition of each transmitting point. Then, the television 202 may beable to receive this data from an application 208 on the user phone, andstore it in repository 104 a. This application, in an embodiment, may bea cloud application 1115 a, as described above.

In particular, because a plurality of WiFi transmitting devices arepresent within the high-resolution zone 212, there is a higher degree ofoverlap of signals allowing for a more detailed and higher resolutioncomposite map to be made. Such a map may be made using the processingpower of a GPU as described above, and may be made also using a cloudnetwork, wherein data from the central module repository 104 a may betransmitted to the cloud network backend resources 1115 for processingas described above. Because as described above the speakers 210 a-210 dmay be present at the same height in an embodiment, the threedimensional composite map made from just these devices would only bepresent in two dimensions in data at a high resolution, since all ofthese speakers are at the same height. Thus the other devicesbroadcasting signals such as WiFi remote 206, center speaker 210 e, userphone 204, and subwoofer 210 f are important as these are at differentheights and aid in sending signal strength at a multitude of differentpoints along the height axis, wherein the composite map can then trulybe constructed in 3 dimensions, with three dimensions of data, combinedwith the high-resolution map in two-dimensions from the speakers 210a-210 d. It is also to be noted that even though the embodiments hereindescribe the use of Wi-Fi signals, other RF signals such as BLUETOOTHsignals can also be used in substantially the same manner.

FIG. 4 shows an embodiment 400 of how presence detection of one or morehumans in front of the TV may occur with the reference setup as shown inFIG. 2 . In particular, the signal strength may be blocked or receivedat a much lower degree due to the presence of a human in the directionof transmission. For example, a scenario when a human A is directly inthe direction of transmission of the Wi-Fi signal from the speaker 410 ato the television 402 can be considered. When this occurs, the RSSI andcorresponding dBm of the signal from 410 a to 402 is lower. Further,there may be software in the RAM or secondary memory 1210 of apparatusmodules 102 to control the wireless network interface 1224 such that thesignal may be boosted, or a signal can be forced to send. This is uniquebecause usually data output from a software application is pushed to astack, where per 802.11 protocols the data packets must then be routedthrough the network interface layer, through the software controllingthe network interface controller, to be sent out wirelessly.

In an embodiment, the television 202 or other receiving entity, if thesignal is detected as lower than normal (e.g. with human A blocking itabove), can send a signal to a transmitting apparatus module (e.g.speaker 410 a) to force-send a boosted signal, wherein the software notonly pushes data but also controls its transmission through the networkinterface layer, and can help expedite sending of such a boosted signalfrom its virtual transmitter as shown in FIG. 3 . Thus when a boostedsignal is sent, the position of 410 a can be verified to a greaterdegree, and data can be sent from the speaker to the TV. Conversely, theTV using such a boosted signal can also send data (e.g. sound) to beplayed by the speakers even though a human A is present directly in thedirection of transmission. In this way, by varying signal strength andtimes from individual transmitters it is possible to pick up the mainsignal and immediately reflected signals from each transmission sourceto provide reliable input data.

In addition to boosted signals, an ad-hoc mode as shown in FIG. 5 mayalso be used to greatly enhance signal reception and can be used as areference point for comparison. In FIG. 4 , as shown, the transmissionfrom all apparatus modules 102 is to one receiving point, television202, only. In this case, speaker 410 a is blocked by human A, and it mayneed to be boosted to get through to the television 202. In contrast, inFIG. 5 , the transmission from speaker 510 a may be received by severalapparatus modules 102 at once. Information regarding the receptionincluding transmitting module, receiving module, RSSI, dBm, direction,accelerometer/gyroscope, other sensor readings, etc., may then betransmitted from the respective receiving apparatus modules to thetelevision 202, which can store this information in the central modulerepository 104 a. Alternatively, the reception information can betransmitted as described above from the receiving apparatus module 102directly to a cloud network, where it can be stored in a repository of acomputing resource 1105. In this manner, many signals from the sametransmitter can be received. As a result, especially within thehigh-resolution detection zone 212, the number of transmissions andreceptions increases rapidly, and allows for an even higher resolutionof Wi-Fi signal strength in three-dimensions to be achieved. Thus, asdescribed above from a variety of apparatus modules 1022, input iscollected in step 909 of FIG. 9 , and stored in the central modulerepository 104 a, at step 911 of FIG. 9 .

From the resolution of detection described in FIGS. 2-5 above, baselinedata may be accumulated (transmission and reception signals when thereare no humans present in the region proximate to the TV as shown in FIG.2 ), and this data as written in the central module repository 104 a, orstored on computing resources 1115, may be compared to the case oftransmission and reception signals when there are humans present in theregion proximate to the TV in step 913 of FIG. 9 . In an embodiment,simple logistical regression or a decision tree may be used, wherein anoverall first score may be computed representing the baseline 3-ddetection of Wi-Fi signal with no humans present. Then, a second sampledetected overall score of three dimensional detection of the Wi-Fisignal may be compared to this first score, and if the ratio of the twomeets a certain threshold below or above 1, a human may be deemed to bepresent.

In an embodiment, further Boolean logic may be used, to account for themovement and handling of a cell phone 204 or a remote 206. For example,an embodiment of the remote 206 is shown in FIG. 8 , with multiplebuttons 806-807 and an IR transmitter where the numeral 802 is located.A baseline score may be present with regard to a composite measurementof movement of the phone detected by the accelerometer, gyroscope, keypresses over a substantial period of time etc., for both the cell phone204 and the remote 206. For the remote 206, the score might also includedetection of IR transmission activity over a period of time by thetelevision 202. Then, when the composite score is above a thresholdcompared to the baseline score, one or more humans may be deemed to bepresent. In this case, if the cell phone 204 of the reference setup 200is detected as tilted to an almost vertical angle when the phone cannotnormally lie vertically in a resting position, and keys are detected aspressed over an extended period of time, it would be determined that ahuman is using the cell phone 204 in front of the TV, and this would bereflected in the measured score compared to the baseline score. The sameis applicable for a remote 206, where for example the lifting up of aremote 206 and rapid key depressing over a period of time (e.g. 2minutes) would normally indicate human use.

Machine learning techniques may be useful as a dataset that isaccumulated for multiple humans in front of the TV, and can capture thetrends associated with features that tend to indicate human presence,and further the number of humans or the type of human present. Inparticular, in this case, received signals form a part of the Wi-Fisignature, e.g. received signals from 210 d, 210 c, 210 b etc., where inthe ad hoc mode, received signal destinations are also recorded (ascompared to the single receiver mode in FIG. 4 , where only television202 is receiving the signals). Additionally, gyroscope and accelerometerreadings from the cellphone, key depressing readings from the cellphone,gyroscope and accelerometer readings from the Wi-Fi remote, all of whichmay be detected in transmissions to television 202, also form featuresof the Wi-Fi signature, and are stored in central module repository 104a. In an embodiment, as shown by element 903 in FIG. 9 , at least one ofthe speakers present in reference setup 200 in FIG. 2 may also have anin-built microphone, and can record and transmit ambient sound over aperiod of time to be stored in the central repository 104 a and alsoform part of the Wi-Fi signature, and can also be included in thebaseline and measured scores described above as a further basis ofcomparison.

In an embodiment a machine learning classifier can also be used forthresholds of presence detection in step 913 of FIG. 9 . In particular,for each element of the Wi-Fi signature, a support vector machine (SVM)could be used to construct an hyper plane in N dimensions between twoclasses (present/absent), indicating the presence or absence of humansin front of the TV, with each dimension corresponding to an aspect ofthe Wi-Fi signature. However, because as described above, it isdetermined that there is a user present in front of the TV when a usercell phone such as 204 or a remote such as 206 is handled in a certainmanner, in an embodiment Boolean logic may be used with three separateSVM's, where there is a SVM for all aspects of the Wi-Fi signatureexcept for those dealing with the cell phone and remote sensor readings,for each of which separate hyperplanes may be created. Then, the resultof all the SVM's could be put together with OR statements, such that ifthe presence of cell phone use in a certain manner or a remote in acertain manner is captured by the SVM, then the result of one or morehumans in front of a TV can be positively indicated.

In an embodiment, a neural network with hidden layers andbackpropagation may be used as a machine learning classifier for theinputs of the Wi-Fi signature, in determining the presence of one ormore humans in front of a television such as 202. By using such aclassification technique, it may be possible to create a system of nodeswith weights. This system of nodes with weights may be used in the togive a reliable prediction, based on the inputs of the Wi-Fi signature,on whether one or more humans are present in front of the television (inthe form of a YES, or a NO response). Based on this predictive analysis,the machine learning logic of the neural network, including the systemand nodes shown in e.g., FIG. 6A., may be executed by the processor ofthe TV central module 104 on the data stored in the central modulerepository 104 a, or alternatively the data stored in the central modulerepository 104 a may be sent to a cloud environment 1101 to be operatedon utilizing multiple computer resource systems 1105. Accordingly, basedon the prediction by such a technique, a decision for the presence orabsence of one or more humans in front of the TV may be rendered in step915 of FIG. 9 .

The different components of the neural network model shown in FIG. 6Awill herein be explained. The input layer 602A contains nodes 1 to i,which represent inputs into the model. Each of these nodes correspondsto a different aspect of the Wi-Fi signature as described above. Forexample, node 1 may represent the signal received from speaker 210 a to210 e in FIG. 2 , node 2 may represent the signal received from speaker210 a to 210 c, and so on. Output layer 606 a may only include twonodes, node 1 and node 2. Node 1 may correspond to a result of ‘YES’ andnode 2 may correspond to a result of ‘NO’ or vice versa, wherein basedon the inputs and weights from each node to the other (w_(ij) as shownin FIG. 6A), the results of the output layer are tabulated, and the node(1 or 2) in the output layer with the greater result is outputted as theoutcome of the predictive analysis.

In traversing from the input layer 602A to the output layer 606A, theremay also be several hidden layers 604A present. The number of hiddenlayers 604A may be preset at one or may be a plurality of layers. If thenumber of hidden layers 604A is one (such as shown in FIG. 6A), thenumber of neurons in the hidden layer may be calculated as the mean ofthe number of neurons in the input and output layers. This is derivedfrom an empirically-based rule of thumb in ease of calculating weightsacross layers. According to an additional rule of thumb, in anembodiment to prevent over-fitting, where the number of neurons in inputlayer 602A is N_(i) and the number of neurons in the output layer isN_(o), and the number of samples in the training data set, from theaggregated metadata is N_(s), then the number of neurons N_(h) in onehidden layer may be kept below

$\begin{matrix}{{N_{h} = \frac{N_{s}}{\left( {\alpha*\left( {N_{i} + N_{o}} \right)} \right)}},} & \left( {{equation}1} \right)\end{matrix}$

where α is a scaling factor (typically ranging from 2-10). In thismanner, the number of free parameters in the model may be limited to asmall portion of the degrees of freedom in the training data, in orderto prevent overfitting.

From the input layer, based on the weights from each node in the inputlayer 602 a to the hidden layer 604 a shown in FIG. 5A, there may be asigmoidal transfer function in going from the input layer 602 a to thehidden layer 604 a. Initially, the weights w_(ij) may be initialized torandom values between 0 and 1. An input signal may then be propagatedaccording to these weights (forward-propagation), wherein the hiddenlayer 604A forms the first outputs for the neurons of the input layer602A. For example, inputs given as neuron 1 and 2 in the input layer602A may be multiplied respectively by weights w₁₁ and w₁₂,respectively, and summed to form the output to the hidden layer 604A.Then the node 1 at the hidden layer 604A may take this net value (Valueof Input Layer Node 1*w₁₁+Value of Input Layer Node 2*w₁₂) and transferthis activation value to see what the neuron output onwards to theoutput layer actually is. At each output layer (hidden layer 604A withrespect to input layer 602A, and output layer 606A with respect tohidden layer 604A) transfer functions comprising the sigmoid activationfunction

${{S(x)} = \frac{1}{1 + e^{- x}}},$

hyperbolic tangent function

${{tanhx} = \frac{e^{2x} - 1}{e^{2x} + 1}},$

or smooth rectified linear unit (SmoothReLU) function ƒ(x)=log(1+e^(x))may be used to transfer outputs.

In the example above, the output given from the input layer 602A toneuron 1 of the hidden layer 405A (Value of Input Layer Node 1*w₁₁+Valueof Input Layer Node 2*w₁₂) would be inputted as the activation value tobe transferred at the hidden layer 604A to one of the transfer functionsdescribed above, and the output would form the value of neuron 1 of thehidden layer 604A to be given onward as input to the output layer 606A,and multiplied by respective weights to the neurons 1 and 2 of theoutput layer. In this manner, full forward propagation of input nodes 1through I in the input layer 602 a may be achieved to the output layer606 a.

Then, to conduct backpropagation, error is calculated between theexpected outputs and the outputs forward propagated from the network. Intraining the neural network, k-fold cross validation, as describedabove, may be used, particularly when the data sets are small. Fork-fold cross-validation, for example, there could be an aggregated setof meta-data of humans present (YES) or not present (NO) with respect todifferent Wi-Fi signatures, comprising all the components describedabove. This set of meta-data may be shuffled and split into a k numberof groups (e.g., 5 groups if k is 5, each holding a particular number ofresults (YES/NO) and corresponding Wi-Fi signatures). Then, for eachunique group, the group can be held out as a test data set, with theremaining groups of aggregated metadata being used to train theclassifier. Finally, based on the training, the accuracy with respect tothe test group can be evaluated. One group may be held for testing andthe others may be used to train the model. In so training, a ‘1’ valueis reserved for neuron 1 and a ‘0’ value is reserved for neuron 2 of theoutput layer 606A if the result is a ‘YES’ for detection of presence ofone or more humans in front of the TV. Conversely a ‘0’ value isreserved for neuron 1 and a ‘0’ value is reserved for neuron 2 if theresult is a ‘NO’ for applicant product eligibility. In this manner,error is calculated between the expected outputs of 1,0 so described,and the outputs actually forward propagated by the network (initially byrandom weights assigned as described above). To transfer the error, theerror signal to propagate backwards through the network is given byerror=(expected−output)*transfer_derivative(output), whereintransfer_derivative is the derivative of the transfer function used(sigmoid, hyperbolic, or SmoothReLU). The error signal for a neuron inthe hidden layer 604A is then calculated as the weighted error of eachneuron in the output layer, according to the weights from the outputlayer to the neuron in the hidden layer 604A. Similarly, the errorsignal from the hidden layer is then propagated back to the input layer602A. Once the errors are calculated for each neuron in the network viathe back propagation method described, the errors are used to update theweights according to the formulanew_weight=old_weight+learning_rate*error*input. Here, the old_weightvariable is the previous given weight in the model, the learning_ratevariable is a value from 0 to 1 that specifies how much to change theold weight to correct for the error, the error variable is the errorcalculated by the backpropagation procedure, and the input variable isthe value of the input that caused the error. Over time, this model canbe developed to form a robust prediction analysis, and the rulesgoverning its nodes, weights, and functions may be written in centralmodule repository 104 a, such that it may be used in step 915 of FIG. 9for accurate detection of human presence.

Furthermore, instead of step 915 in FIG. 9 , a multitude of machinelearning classifying techniques described above may be combined to forma composite machine learning classifier as shown in FIG. 6B. Suchtechniques may include SVM, random-forest, decision trees, neuralnetwork, etc. As shown in this figure the aggregated metadata, whichforms the input data 602 b, may be inputted into multiple differentmachine learning classifiers 604 b 1, 604 b 2, etc. A combining decisionrule 606, which may combine votes from multiple classifiers, can thenfinally decide a class label (YES or NO for eligibility) in step 608,which can be returned as the outcome from the ensemble classifier. Anexample combining decision rule 606 may be a majority vote rule.Especially when models are equally well performing, such a combiningdecision rule can help balance out their individual weaknesses, and canmake for a more robust classifier overall.

In terms of the aggregated metadata and parameters used in the machinelearning or comparative/logistic regression models described above, thisalso may be gathered or set in a number of different ways. Firstly,there may be an embodiment where the parameters have already beenextensively tested for by domain experts in a reference set up such asshown in FIG. 2 , with a predetermined orientation and spacing ofapparatus modules 102 relative to a central module television 104. In anembodiment, these parameters may be hardcoded.

Secondly, parameters may be learned from manual labeled data, that iscollected in live-user tests, where conditions are rigidly controlledfor, such that e.g., ambient noise isn't due to a dog nearby, or a birdcrashing into the window of a set-up, but rather clearly helps compare abaseline situation with when a human is present.

In both embodiments above, whether by domain experts or learned frommanual labeled data, this can supplement learning parameters by testingon individual user data. Alternatively, only testing on individual userdata can be used. To accurately test on user data, the display of thetelevision, as shown, e.g., in 402 a of FIG. 4 may display the GUI shownin FIG. 7A from time to time, or at predetermined intervals. The user,using the remote 406 as shown in FIG. 8 , with yes/no buttons 804 and806, respectively, may be able to respond to such a prompt and create adata point corresponding to a Wi-Fi signature. Elements 810, 812, and814 in FIG. 8 represent internal components of the remote, including theWiFi transmission module, accelerometer, and gyroscope, respectively.

The same GUI in FIG. 7A may alternatively be able to be displayed in anapplication on the user's phone, prompted by the television 402, whereinsuch an application may be of the type of cloud application 1105 a asdescribed above. In this case too, the user's response in terms of yesor no would be able to be received by the central module 104, with theresult recorded in the central module repository 104 a, creating a datapoint corresponding to a particular Wi-Fi signature. In this manner,over time, a vast data set comprising aggregated training metadata canbe accumulated. For even more accuracy in training, the datasets ofmultiple users, with their permission, may be able to be shared in thecloud over a cloud network 1101, wherein computing resources 1105 canstore and aggregate data from multiple users in databases and canfurther develop models to fine tune distinguishing of human presence infront of a TV.

Finally, after the presence or absence of one or humans in front of a TVhas been determined by steps 913/915 of FIG. 9 , based on the presenceof a user detected at step 1001 of FIG. 10 , further actions may betaken by the IoT (Internet of Things) system comprising the centralmodule 104 and the apparatus modules 102. In particular, if one or morehumans are detected in front of the TV, at step 1005 the systemdetermines if media is being heard or watched. In particular, atelevision such as television 202 has the capability of knowing whichprogram is being watched, and thus is able to determine whether media isbeing watched on the TV. Similarly, using the internal microphone in atleast one of the speakers shown in FIG. 2 to record ambient sound, itcan be determined whether the user is listening to media through thespeakers (e.g. through a BLUETOOTH or wireless connection). In thiscase, if it is determined the user is consuming media, based on theabove detection of the Wi-Fi signal in three dimensions a location wherethe user, or users, might be present, is identified through drops in thesignal compared to a baseline measurement. Once such a position isidentified, the device remote 206 may enter a full-power mode so thatthe user may maintain full use of the remote, and the system may enter ahome theater mode. In this mode the IoT system may adjust severalcomponents to provide an optimum experience to the user based on theirlocation. For example, in FIG. 4 , the speakers 410 a-410 f and lights412 a-412 b may be on a stage that is controllable in the up-to-down andleft-to-right directions. This stage may be able to be controlled by theIoT system, wherein a signal sent by television 402 to the speakers andthe lights to be arranged in such a position that sound reaches users Aand B equally in all directions.

This can further be customized by the user, for example, through anapplication connecting the television 402 to the phone 404, such as anapplication 1005 a, wherein the user can specify they would like toadjust bass, treble, or other sound settings. For example, if the usersA and B would like to enter a deep bass mode, they can enter such a modeby pressing a yes/no key similar to 705 a and 707 a, on a questionprompt similar to that of GUI 700 a, instead asking “Do you want toenter a bass mode?” If the response to such a prompt is Yes by the user,then the bass sound on the rear speakers 410 d and 410 a may beincreased by the television 402 relative to location where users A and Bare present. Instead of buttons yes and no 705 a and 707 a, slidersmight also be included to adjust the bass/treble, or volume levels, etc.

As an alternative or additionally, if the television detects. e.g., thatthe user is watching a TV show or a movie, the television may instructthe lights 412 a-412 b to adjust position in the up-to-down orleft-to-right direction, and may change the hue of or the intensity ofsuch lights to provide an optimal viewing experience for the user. Theuser may be additionally able to choose his or her preferred color orintensity through sliders similar to those mentioned above in thelocations of 705 a and 707 a on a GUI similar to 700 a. In addition, inanother embodiment where speakers or apparatus modules 102 are movablemanually, a graphic might be displayed on the television screen such as404 a or on such an app 1005 a, instructing the user relative to theirlocation where they can move speakers for better sound quality.

Furthermore, based on the detected content being watch, there IoT systemmay use a further machine learning model to assign contentrecommendations to certain users. In particular, a prompt such as FIG.7A may pop up on the television screen, or on the screen of a user phone204 using an app such as 1005 a connecting the phone 204 with thetelevision 202, asking “Are you a new user?” If the user clicks yesunder 705 a using either the remote 206 or the phone 204 as input, thenthe television may guide the user through a registration process andstart associating content watched with a particular user, as well asWi-Fi signatures with a user. For example, many humans have differentfavorite poses or gestures that may be unique to them, such as lyingdown on a sofa such as sofa 214 shown in FIG. 2 , or sitting upright, orpicking their nose, or nervously biting their teeth, etc. The monitoringof the Wi-Fi signature by gathering input from the environment and userat step 909 may occur at regular intervals, and for a continuouspredetermined period of time (e.g. anywhere between 1 second-10minutes), and this process may occur even after the presence of one ormore humans has been conducted in step 1001.

That is, at step 1017 through monitoring of Wi-Fi signature associatedwith registered users, additional aggregated metadata can be gatheredwith for an output class of people. Thus, a machine learning model,including a deep learning neural network model with back-propagationsimilar to step 915 may be able to be used. However in this model, theoutput layer would not simply be two nodes representing YES or NO withrespect to detection of the presence of one or more humans, but wouldrather be node 1 representing Registered User 1, node B representingRegistered User 2, etc., where there may be a plurality of registeredusers. By adding metadata of content watched as detected by the TV afterstep 1005, and monitoring for change in Wi-Fi signature that may beassociated with physical user poses, the machine learning model may beable to associate certain TV shows as input with respective users.

Accordingly, for all users it may be able to provide contentrecommendations in line with what they typically have liked on theirmetadata. Furthermore, through association of poses through change inWiFi signature with certain registered users as input in the model, whena user is even detected to be sitting or making a unique pose on a sofa414, this may enable the model to determine that a certain registereduser is sitting on the sofa in front of the TV. In particular, the modelis trained to adjust the weights of input nodes such that a change ininput components into the machine learning model comprising the WiFisignature reflected by making such a pose enables the model to determinethat a certain registered user is sitting on the sofa in front of theTV.

In addition to the inputs previously mentioned, variance of said inputs(e.g. variance over a predetermined period of time of transmission ofWi-Fi signal from speaker 210 a, etc.) may also be used as an additionalor alternative input in the input layer 602 a of the machine learningmodel. Then, when the model determines a certain user has sat in frontof the TV, the television 202 may turn on automatically the respectiveuser's favorite show, or a show it thinks the user would like based ontheir aggregated metadata.

For example, often when a user may arrive home after a busy workday,he/she may not have the energy to turn on the TV. In this case the modelcan turn his/her favorite TV show on automatically. To make sure thatthe TV is collecting metadata for the correct user, it can ask from timeto time, in a prompt similar to GUI 700 a “Are you user X,” where user Xreflects the user the model determines may be sitting in front of theTV. The Yes/No 705 a, 707 a response by the user can further serve astraining data for the model. In this manner, when minors are watching TVthis can also be detected by such a machine learning model, and whensuch a determination is made, parental controls can be turned onautomatically by the TV, such as 202, to prevent minors from watchingcontent that may not be suitable for them.

The IoT system comprised of the apparatus modules 102 and the central TVmodule 104 further comprises a suite of environmental and safetyfeatures with regard to the reference set-up 200 shown in FIG. 2 .

First, at step 1005, as previously discussed above, media being heard orwatched is detected. In this case, if no media is being heard orwatched, then the system may keep monitoring input at predeterminedintervals from its constituent components as discussed above. After apredetermined amount of time, it can enter a green mode to turn thetelevision off. Even in the case that it is being watched, an additionalmachine learning model can be used to identify changes in Wi-Fisignature, as discussed above, to particularly determine when a user maybe sleeping even though media is playing. In this case, the systementers the green mode. In the green mode, the system may also controlbattery-powered devices such as the device remote 206, or any of thespeakers 210 if they are battery powered (such as the subwoofer 210 fwhich may need extra batteries to amplify wattage), to enter a low-powermode in order to conserve battery power.

Particular poses are identified with sleeping, in which case the IoTsystem transitions from YES at step 1005 to step 1009, in variance withthe normal procedure described above, where the system would transitionto steps 1011 and 1017. In this case, because the system is continuouslygathering input and storing it in steps 909 and 911, periodically adetection for sleep may be run on this data. In particular, an inputchange in the Wi-Fi signature in three-dimensions, including ambientsound recorded as described above, may be unique for a change such aslying down, resting one's head on a pillow, or even leaning back on asofa such as 214.

In addition, sounds such as snoring that are uniquely associated withsleeping may be within a certain frequency and discernable by theambient sound recorded which is also an input into the model. Because itis difficult to ask a user when he/she is sleeping to generate metadata,pre-generated meta-data from user tests or experts. Then using theseparameters, and the previous input as discussed into the input layer 602a of a machine learning model such as the neural network withback-propagation as shown in 6A, the output layer 606 a would berepresentative of two nodes, 1 which would represent that the user issleeping, and 2 which would represent where the user is not sleeping. Ifthe value of 1 after the model had been propagated would be greater than2, then it would be determined that the user is sleeping.

This is shown as well in step 1015. Although such monitoring for sleepthrough machine learning could also occur periodically (such as evenafter YES is determined at step 1005 as described above), step 1015shows that it would also occur even after the television/media has beenturned off. That is, in an embodiment, even after the television/mediais turned off, input continues to be gathered in steps 909 and 911, onthe basis of which monitoring for sleep continues to occur at step 1015.

When a user is detected as asleep in front of the TV, the IoT system mayadjust accordingly in step 1023, by entering sleep mode. For example, asdescribed above, it may control the movement of speakers and lights insuch a manner as to dim lights (e.g. 412 a and 412 b) to play ambientmusic conducive to deep sleep to provide an adequate sleepingenvironment. The user can change any of these preferences by accessingan appropriate menu in the TV. Further, a user may be able to pre-set analarm using an application of a type such as 1005 a from his/her phone404 in communication with the TV, or using an application stored in thememory of the TV 402 along with the remote 406. In particular the usermay set a desired time using a numeric keypad 808, as shown in adetailed view of such a remote in FIG. 8 , or by pressing keys onhis/her cell phone 404, which can then output and received by the TV402.

The TV can then perform an ambient sunrise as shown in 402 a of FIG. 4 .The sunrise may begin a predetermined amount of time before the user hasrequested to be awoken up, so as to be woken up gently. In particular,the TV screen of a TV central module 104 may display graphics (e.g. aSun moving), executed by the GPU or by the processor of the TV describedabove. These graphics gradually become bright in the direction that theuser is sleeping (wherein the user's position may be determine, e.g., asexplained above). In addition, ambient sound can gradually get larger.Determination of if the user is still sleeping, as with the machinelearning model in step 1015 can continuously be made. In case the useris still sleeping at the time he requested to be woken up, loud musiccan be played particularly in speakers proximate to the user'sdetermined location, so as to wake the user up quickly. In this manner,a sleep mode and waking up routine in 1023 can be performed by the IoTsystem.

Finally, the IoT system is equipped with the safety feature of detectingan intruder. In step 1003, if at step 1001 the presence of a user is notdetected in front of the TV, then a further machine learning model, asdisplayed in 6A or 6B may be run on gathered data to check if there isan apparent intruder outside of the zone of detection shown in FIG. 2 .In an embodiment, such a machine learning model may continuously bemonitored for a positive result (indicating there is an apparentintruder), even if the presence of a user is detected in 1001. In suchan embodiment the detection of step 1003, as with step 1015, may occurat continuous predetermined intervals (e.g. anywhere between every 1second-10 minutes).

In particular, unique changes in the Wi-Fi signature, defined by some orall of the inputs described above, may be associated with the presenceof an intruder. For example, furtive fighting motions may have aparticular Wi-Fi signature in three-dimensions or cause a unique changeof inputs of the Wi-Fi signature over time. At first, the parameters forsuch a machine model may be hardcoded by experts identifying uniquechanges in Wi-Fi signature. For example, since the ambient sound pickedup by at least one microphone as described above is an input in theinput layer 602 of such a model, the sound of shattered glass indicatinga door breaking may be programmed to be recognized as associated with aburglar. Similar sounds might include people screaming, gunshots beingfired, etc. The output layer 606 a of such a machine learning modelincludes two nodes, where node 1 may represent an intruder beingpresent, and node 2 may represent no intruder being present.

As described above, if the value of node 1 after propagation is lessthan node 2, then the output from the model may be that there is not anapparent intruder (NO at step 1003). In this case, the same steps asdescribed above with the green mode (1009 to 1015 to 1023) may befollowed by the system.

Alternatively, however, if the value of node 1 is greater than node 2,then the model may output an apparent intruder present (YES at step1003). In this case, the location of the intruder is determined. Ifsounds and movement are determined to be coming from outside the room inwhich the sensors are located (or outside, e.g., the high resolutiondetection zone 212), then GUI 700 b may be displayed on the TV screen(in case there is a registered user at home) and simultaneously sent toa registered user's phone, using an application such as 1005 a describedabove connecting the phone such as 204 with the TV 202.

When the phone 204 is carried away from the detection zone of the TV202, such a message can be sent from the TV 202 wireless networkinterface 1224, over the internet, and to the phone 204. The message canbe programmed to flash on the user's phone screen, wherein the user musteither choose an option 705 b to sound an alarm at the house, or anoption 707 b to ignore the warning. Meanwhile, the TV module (e.g. 202)may automatically turn on a play content to make a potential intruderbelieve there are guests inside the house, as a deterrent.Alternatively, if sounds and movement are determined to be coming insidethe room in which the sensors located (e.g. inside the high resolutiondetection zone 212), then because the IoT model has detected an insideintruder, the message is only sent to a registered user's phone asdescribed.

Then, after a predetermined period of time after the GUI has beendisplayed, if a user response has been determined, corresponding actioncan be taken. In the case of ignore 707 b, under step 1019, no action istaken. In the case of sound alarm 705 b, under step 1019 very loudsounds are played on the TV, and the speakers can be programmed torecord any events occurring in the vicinity of the setup. In case therehas been no response from the user, by default the TV may play loudsounds or activate a decoy mode as described above to make a potentialintruder believe there are guests inside the house, under step 1021. Theuser can choose which of the two options under step 1021 he prefersthrough a menu of the TV module 202, or one of the two options can bechosen at random. Based on the type of response received from the userate step 1013, or no response received, in which case an intruder ismarked as present by default, metadata is collected and a data point isstored in the central module repository 104 a. In this manner, datacollected from multiple users may be shared across computing resources1105, and the nodes and weights of such a model can be better trained.

Embodiments describing further use of the WiFi remote 206 forenergy-efficient system operation, and for boosting presence detectioncapabilities of the system as a whole, will herein be described. It isnot always the case that speakers may be present in the system, or thatthe user will necessarily be present in the detection zone 1312 createdby such speakers.

FIG. 13 shows a reference setup 1300 analogous to reference setup 200 ofFIG. 2 , but wherein a user 1316 may be present in a zone 1320 depictedby shaded lines that is entirely outside of the high-resolutiondetection zone. For example, a user 1316 may be sitting at a coffee ordining table with a WiFi remote 1306. In this case, the user may be in acold zone, wherein the system may keep track of zones within apredetermined geographical region 1318. The system, in central datarepository 104 a, may have a composite map of regions as describedabove, wherein in such map there are hot zones that arewell-triangulated for detection with multiple sensors, such as thedetection zone 1312 created by the presence of the speakers 1310 a-1310f, wherein regions falling outside of these zones in the map aredesignated cold zones.

In a further embodiment, the speakers 1310 a-1310 f may not even bepresent at all. In either embodiment, the user 1316 is in an area thatis a cold zone and not well-triangulated. In this case, the user maystill have the WiFi remote device 1306. For example, the user may behaving a meal or a drink, and would like to switch channels and watchthe television 1302 from his or her location in the cold zone 1320.

Due to its proximity to the user when the television is turned on basedon its most frequent uses (to change channels, browse menus on thetelevision screen, etc.), the device remote 1306 provides a very highquality link for detecting the presence of a user. There are severalways in which the remote 1306 can work in unison with the television1302 and other speakers 1310 (if they are present) to determine thepresence/absence of a user as well as the geographic location of theuser in the predetermined geographic zone 1318.

One way in which the absence or presence of a user may be immediatelyevident is through user-based manipulation and activity of holding andpushing buttons on the remote 206. As described above with respect toremote 206. Boolean logic may be used, to account for the movement andhandling of remote 1306. For example, an embodiment of the remote 1306is shown in FIG. 8 , with multiple buttons 806-807 and an IR transmitterwhere the numeral 802 is located. A baseline score may be present withregard to a composite measurement of movement of the remote detected bythe accelerometer 812, gyroscope 814, key presses over a substantialperiod of time etc. The score can also include detection of IRtransmission activity from the transmitter 802 over a period of time bythe television 202. Then, when the composite score is above a thresholdcompared to the baseline score, one or more humans may be deemed to bepresent. Further, if the remote 1306 of the reference setup 1300 isdetected by the gyroscope 814 as tilted to an almost vertical angle whenthe remote cannot normally lie vertically in a resting position, and theaccelerometer 812 detects accelerating movement wherein keys 808 arealso detected as pressed over an extended period of time, it would bedetermined that a human is using the remote in front of the TV, and thiswould be reflected in the measured score compared to the baseline score.

Geographic location may be determined by WiFi signals transmitted fromthe remote's WiFi radio 810. In an embodiment, through a requestdisplayed on the television or through a mobile application on the userphone 1304, the user may be asked to walk around the cold zone 1320. Inthe embodiment above where the speakers are not present, this cold zone1320 may comprise the entirety of the area of 1318 in front of thetelevision 1302 and router 1324. The router 1324 may comprise anadditional apparatus module 102 in communication with the television1302, where apparatus modules 102 have been detailed above.

Without the presence of the remote 1306, the detection of human presencemay be only from the WiFi radio of the television 1302, whichcommunicates with the router 1324. This mode of detection may be ratherlimited. In particular, if the user is located directly between thetelevision 1302 and the router 1324, which is a small segment of FIG. 13as indicated by the dashed bidirectional line between the twocomponents, then the television may be able to detect the presence ofthe user with a higher degree of accuracy. In this case, because theuser is present directly in the transmission path from the WiFitransmitter of the television and the WiFi receiver of the router 1324,the presence of the sure would result in a significant drop in signalsuch that detection with a reasonable degree of certainty could occur.Because the antenna of television 1302 and router 1324, respectively, isomnidirectional, there are additional transmission paths in alldirections, where for instance the signal may be reflected off of thewalls of geographic zone 1318 and received by the router 1324, and viceversa. However, because of loss of signal upon reflection, increaseddistance to the router, etc., any blockage in these additional paths mayresult in a lower impact in the overall signal received by the router (adrop in signal may be small and not able to be easily detected withcertainty).

For example, in the case as shown in FIG. 13 , the user 1316 is locatedsubstantially far away from the transmission path between the television1302 and the router 1324. There may be a transmission path wherein WiFisignal transmitted from the router 1324 may reflect off of windows 1322and return to the television 1302, wherein the presence of a user 1316may result in the blockage of this path and a subsequent drop in theoverall WiFi signal received by the television 1302. However, because ofthe signal loss due to reflection upon encountering the windows, and theincreased distance from the router 1324 to windows 1322 and then fromthe windows 1322 to the television 1302, this transmission path may notcontribute significantly enough to the overall signal received by therouter 1324 for the overall signal received by the television 1302 to bedropped enough that the presence of a user 1316 can be accuratelydetected. Other factors such as general noise in reception, etc., mayalso contribute to small recurrent drops in overall signal received bythe television 1302, where it may be difficult to determine withcertainty whether the drop in signal has indeed been caused by thepresence of a user.

It is in this type of case that it is tremendously useful to utilize theremote 1306 for the use of human presence detection. Because when atelevision 1302 is operated by the user 1316 the user typically utilizesthe remote for operation of the television, the remote 1306 willtypically be located near the user 1316. As a result, it is more likelythat the user will be present to a greater degree in collectively all ofthe transmission paths from the omnidirectional antenna of the WiFitransmitter module 810 of the remote 1306 and the WiFi receiver modulesin television 1302 as well as router 1324. In addition, when the usermay be prompted to walk around the room as described above with respectto geographical region 1318 to better map cold zones, the infraredmodule 802 or the WiFi transmitter module 810 may be used to map otherblockages in sending of the signal. For example, the user may beprompted to move the remote 1306 up and down at various points in coldzones such as 1320 to monitor blockages in the Z or vertical directionof a standard XYZ coordinate system. In this manner, blockages such asfurniture, lamp poles, tables, and other fixtures that may be presentthroughout the room may be accounted for. In particular,three-dimensional coordinates of hot and cold zones, as well aslocations where potential fixtures may be present, can be mapped in thismanner to the composite map. The infrared module 802 of the remote maybe particularly useful wherein furniture is present between the remote1306 and a reflective surface that the remote is pointing towards. Forexample, in FIG. 13 , if the remote was moved up and down in itslocation, and a coffee table is present between the remote and the upperwindow 1322 shown in cold zone 1320, then it would be accuratelydetected by the infrared module. On the other hand, when furniture ispresent between the WiFi remote and the television, such as speaker 1310c, with no reflective surfaces in the transmission path, this may bedetected well by the WiFi module 810. In this manner, interpolation ofthe infrared module and WiFi module readings can be used in anycombination to detect for potential furniture present in thegeographical region of 1318, and these potential locations and 3Dcoordinates of such blockages can be added to the hot and cold zones(also in three dimensions) to the composite map of the centralrepository 104 a, where the data of the hot and cold zones may indicatehow weak signal transmission is relative to a baseline from respectivelocations in the composite map, including all potential blockagelocations.

As a result of the locations of blockages already being present in thecomposite map, because of the accelerometer and gyroscope 812 and 814located on the remote 1306, it has the positioning capability, togetherwith WiFi signal triangulation (occurring as described above) from therouter 1324 and television 1302, to locate where the remote is presentwith respect to the composite map. Such WiFi signal triangulation may beused in conjunction with triangulation data from the speakers asdescribed above for a more accurate representation, or can be used aloneif no speakers 1310 are present. Therefore, if the remote is in alocation where furniture may be located in the transmission paths fromthe remote 1306 to the television 1302 or the router 1324 such that itmay result in a substantial drop of signal, the signal to be sent fromthe remote 1306 at that location can be boosted beforehand by the CPU ofthe central module 104, such that the signal can overcome the drop inreception due to the furniture, and can still be used to detect thepresence of a user that may be present in such a location. Inparticular, there is a roughly linear correspondence between thetransmission level of the WiFi radio 810 of the remote 1306, and arespective reception level of the WiFi radio of the television 1302 orthe router 1324. Therefore, where a mapped furniture may be blocking thetransmission path of the WiFi remote 1306 to the television 1302 orrouter 1324, and the drop in signal reception relative to a baseline isknown, the transmitted signal from the remote 1306 may be increased bythe same proportion as the drop in signal reception to overcome the dropin reception caused by the presence of the blockage.

This sort of calibration helps the system to adapt in a uniform mannerto different levels of transmission blockages caused by the arrangementof fixtures in a room. In particular, because the relative strength of auser's phone 1308 may vary, and because the signal strength oftransmission of the WiFi may not be able to be controlled by an app onthe user's phone 1308, it is useful to conduct such a mapping, andsubsequent signal manipulation by the WiFi remote 1306. As detailedabove, in this case the CPU of the television 1302 can send to atransmitting apparatus module (here the WiFi module 810 of the remote1306) instructions to force-send a boosted signal, wherein the softwarenot only pushes data but also controls its transmission through thenetwork interface layer, and can help expedite sending of such a boostedsignal from its virtual transmitter as shown in FIG. 3 .

In addition to the zonal and object mapping features of the remotedescribed above, the remote may also offer energy-saving features as ithas two-tiered mode of operation, in a low-power mode, or a full-powermode in conjunction with the low-power (sleep) or regular modes ofoperation of the television 1302. These modes of operation will hereinbe described, wherein the presence detection feature is maintained infull capacity even while these devices may be operating in reduced powermodes. Such energy-efficient modes of operation may be particularlyuseful in the case where the remote is battery-powered, directlyresulting in less expenditure on batteries for consumers. In addition,these modes result in the television and remote together consuming lesspower, which can also lead to less mains power consumed, resulting insavings for a user over the long run. By default, unless the presence ofa user is potentially detected, the remote 1306 may remain in alow-power mode in order to conserve battery power.

Example energy-efficient modes of operation are shown in FIG. 14 . In anembodiment, the mode in which energy-efficient operation of the remoteis conducted may depend on whether the television (TV) 1302 is on or off(as shown in step 1401). Alternately, the same mode can also always befollowed regardless of whether the TV is on or off.

As shown in FIG. 14 , at step 1401 it is determined whether the TV is onor off. Such a determination may occur through the CPU 1204 oftelevision 1302, where standard operating systems may support a numberof system power states, etc., and the status of whether the television1302 is on or not may be stored in a CPU variable, where the CPU may notfully power off although the TV is turned off. Alternately, such adetermination may also occur through the pressing of buttons or handlingof a remote as described above, by detecting sound transmitted from theTV 1302 to any speakers 1310 (if present), etc.

Then, in step 1405, if the TV 1302 is determined to be on (ON as shownin FIG. 14 ), the WiFi radio transmitter present in the TV 1302 may beused to initially scan for the potential presence of a user. This may beuseful to do when the TV is already turned on as determined in step1401, since because the TV is fully powered on, it may not cost anyadditional energy to use the TV WiFi radio, whereas when the TV isturned off, it may take additional energy to first power the TV on in adifferent system power state to be able to use the WiFi radio of the TV.

The WiFi radio of the TV 1302 as discussed above, may not be as accurateused alone to definitely state whether a user is present or not as theresulting drop in signal may also be due to noise, etc., but it canstill be used as a potential indicator for presence. Thus data readingsare first scanned from the WiFi signal transmitter present in the TV1302. Since the TV is turned on, the network layer of the WiFi modulecan be used to force-send a strong signal, for a potentially higher RSSIwhich is beneficial as described above. The signal received by therouter 1324, and vice versa from the router 1324 to the TV 1302, may begathered as raw data by the CPU.

The raw data, gathered in step 1405, may in turn, may be fed to a neuralnetwork machine learning classifier of the type of FIG. 6A describedabove. Such a machine learning classifier may then be used, in step1409, to determine the absence or presence of a user in front of the TVwithin the geographical range of 1318. The CPU may feed the RSSI signalstrength received by the router, the strength of WiFi signal transmittedfrom the TV 1302, and a baseline value based on the composite map orhistorical or other user data for the RSSI strength of a signal receivedby a router at approximately the same strength the signal is transmittedby the WiFi module of the TV in a standard room of approximately thesame size to the classifier. In particular, any combination of thesethree values may be used as values of input layer nodes in input layer602 a of the machine learning classifier of step 1409. After comparingoutput layer node values as discussed above, the outcome of step 1409may be determined by the CPU, indicating whether there is a potentialpresence of a user or not. Such a model may be trained over time. Theresults of a more accurate detection, which occur in step 1419 asdetailed below, may be used to compute error and conduct backpropagationas described above, in the machine learning classifier model of step1409. Thus over time, the model may become more accurate to a degree indetermining the potential presence of a user.

After step 1409, if it is determined that a user is not potentiallypresent (NO at step 1409), then the process simply goes back to step1405 and continues to potentially scan for presence at periodicintervals. On the other hand, if it is determined that a user ispotentially present (YES at step 1409), then the remote 1306 may befully powered up at step 1415. The remote is fully powered up in orderto perform a full presence detection in step 1417.

In this step 1417, the remote 1306 determines potential presence in amore robust and stronger detection mode. At the very minimum, for thereasons discussed above, it is likely that the remote 1306 is proximateto the user 1316 and for that reason provides a high quality link indetermining true presence/absence of a user. As discussed above, thedrop in a WiFi signal transmitted from the remote 1306 to the TV 1302 orthe router 1324 will be more substantial if the user is (as is likelythe case) proximate to the remote. Furthermore, because there are atleast 3 WiFi access points that can all transmit and receive signals(the remote 1306, the TV 1302, and the router 1324), triangulation ofWiFi signals can occur. Such triangulation enables a greater accuracynot only of presence detection, but also of determining a position inthe room as described above. Such a determined position of a drop insignal may also be cross-verified with the composite map to make sure adrop in signal is not a false positive when there is already apre-existing furniture or other blockage at that position in the room.

Such a check with the composite map may involve, e.g., calculating thesignal strength transmitted by the remote 1306 subtracted by the RSSIsignal strength received for either the TV 1302 or the router 1324,further subtracted by any signal drop due to pre-existing furniture inthe same location present in the composite map. If the result is asubstantial amount (not close to zero), then the process may continue tostep 1419. If the result is approximately close to zero (within 0.5 dBmor any other such predetermined interval), then step 1419 is skipped andthe outcome is determined as absent.

Upon cross-checking with the composite map, the raw data gathered fromthe presence detection step 1417 may be inputted into a neural networkmachine learning classifier of the type shown in FIG. 6A. The raw datamay be inputted in the form of signal strength transmitted. RSSI signalstrength received by the TV 1302, RSSI signal strength received by therouter 1324, RSSI signal strength received by speaker(s) 1310 a-1310 f(if they are present), and any pre-existing drop in signal strength dueto nearby pre-existing furniture from the composite map. As describedabove, the data may then be forward-propagated through the model, and anoutcome of presence or absence can be determined. As further discussedabove with reference to the classifier of FIG. 6A, actual results can beused (knowledge of presence/absence of a user or affirmation by a usere.g. through the GUI in FIG. 7A) to calculate error and backpropagatethrough the model, making the model more robust over time. This isreflected in step 1421 of the process. That is, in step 1421, themodel's outputs in step 1419 of presence or absence may also be fed asthe correct result into the separate neural network machine learningclassifier of step 1409 (or of step 1413 as will be discussed below),for calculation of error and backpropagation in that machine learningclassifier. Because of different inputs, different degrees of accuracy,different WiFi radios used for transmission, etc., it is necessary touse different neural network machine learning classifiers at steps 1409(and analogously step 1413 as will be discussed) and step 1419.

The energy-efficient process when the TV 1302 is determined to be off atstep 1401 is now discussed. In this case, if the TV is determined as off(OFF at step 1401), then although the TV is turned off, the CPU maystill be active. In this reduced power state, the CPU can send a lowpower WiFi beacon to the remote 1306.

This is shown in step 1403 of FIG. 14 . Once such a low power WiFibeacon reaches the remote 1306, then the remote may enter a low-powereddetection mode in step 1407. In essence, because as described above thesignal strength in the network layer of the WiFi module 810 of theremote 1306 can be controlled, a signal transmitted in a low-powereddetection mode may be limited to some percentage threshold value (e.g.less than or equal to 40% of normal signal transmission power). Then,once the remote 1306 has entered the low-powered detection mode, whichmay be an intermediate power state between the low-power mode and thefull-power mode of the remote 1306, a low-power presence detection maybe performed by the remote in step 1411.

The low-power presence detection mode is analogous to the full-presencedetection mode discussed in step 1417 above. However, as discussedabove, in the low-power detection mode, the signal transmitted by theremote 1306 is at a lower power threshold. As a result, the result ofsuch a detection, while not conclusively accurate, may be accurate to adegree for detecting a potential presence in step 1413, as with step1409. In particular, raw data is first gathered by the TV 1302 or therouter 1324 at a low-power transmitted signal from the WiFi transmitterof 1306 at step 1411. Then this raw data may be fed to the neuralnetwork machine learning classifier at step 1413 in the same manner asraw data is fed for 1417. In this low-power mode, as with the classifierof step 1417, triangulation may be used in conjunction with the TV 1302and the router 1324 as described above to improve accuracy of results indetecting a potential presence while still operating at low power.

In an alternate embodiment, if the classifier is found to producesatisfactory results even without triangulation, the neural networkmachine learning classifier of 1413 can also be used withouttriangulation present in the model of step 1417. Accordingly, acomparison with a composite map or input of readings from the compositemap into the neural network machine learning classifier of 1413 may notoccur for the embodiment without triangulation. The nodes of the outputlayer of the machine learning classifier of 1413 are interpreted asdescribed above with respect to FIG. 6A for determining the presence orabsence of a user in front of the TV 1302 in the geographical region of1318.

If the outcome of the machine learning classifier at step 1413 indicatesthat there is a user potentially present, then as with step 1409described above, the process proceeds to steps 1415 through 1421. If theoutcome of the machine learning classifier at step 1413 indicates thatthere is not a user potentially present, then the process returns tostep 1403, wherein the CPU operates to make the television 1302 continueto send low power WiFi beacons from the TV to the remote at periodicintervals for low-power presence detection at step 1411.

An embodiment of error correction for the neural network machinelearning classifiers of step 1413 is further described. In particular,because the power is limited to below a certain threshold, a lossfunction (for calculation of error, as described with respect to theneural network machine learning classifier of FIG. 6A above) of theneural network machine learning classifier may be set as a modifiedweighted cross-entropy loss function. This function may be representedin mathematical form as:

${- \frac{1}{M}}{\sum_{m = 1}^{M}{\left\lbrack {{w_{mcfn}*y_{m}*{\log\left( {h_{\theta}\left( x_{m} \right)} \right)}} + {w_{mcfp}*\left( {1 - y_{m}} \right)*{\log\left( {1 - {h_{\theta}\left( x_{m} \right)}} \right)}}} \right\rbrack.}}$

In this equation, M may represent a number of training examples atdifferent levels of power transmission (e.g. one for every percent from1-40%, where a 40% threshold of full power as discussed above may beused), w_(mcfn) may represent a marginal cost of a false negative over atrue positive (absence when there is presence), and w_(mcfp) mayrepresent the marginal cost of a false positive over a true negative(presence when there is absence), y_(m) may represent the label ofpresence of a user, and 1−y_(m) may represent absence of a user, x_(m)may represent input nodes of a training example m, and h_(θ) mayrepresent the model with weights θ. Because the situation of absencewhen there is presence is to be avoided at all costs (the remote shouldbe at full power when a present user wants to use it), the marginal costof a false negative over a true positive may be weighted higher than themarginal cost of a false positive over a true negative. Even if a falsepositive occurs, at the later full presence detection stage it is likelythat an absence will be accurately detected, so little harm occurs.

Furthermore, for progressively higher samples m with higher levels ofpower transmission, both the above-mentioned marginal costs may beweighted higher, because since more progressively more energy is used,more battery power is lost. Thus, using the above modified equation as aloss function in error calculation, and then using the results tobackpropagate through the machine learning classifier of step 1413 mayresult in an optimized classifier with a high degree of low-powerpresence detection (although lower in accuracy than the classifier of1417) while consuming a low amount of power.

While this disclosure describes exemplary embodiments for exemplaryfields and applications, it should be understood that the disclosure isnot limited thereto. Other embodiments and modifications thereto arepossible, and are within the scope and spirit of this disclosure. Forexample, and without limiting the generality of this paragraph,embodiments are not limited to the software, hardware, firmware, and/orentities illustrated in the figures and/or described herein. Further,embodiments (whether or not explicitly described herein) havesignificant utility to fields and applications beyond the examplesdescribed herein.

Embodiments have been described herein with the aid of functionalbuilding blocks illustrating the implementation of specified functionsand relationships thereof. The boundaries of these functional buildingblocks have been arbitrarily defined herein for the convenience of thedescription. Alternate boundaries can be defined as long as thespecified functions and relationships (or equivalents thereof) areappropriately performed. Also, alternative embodiments can performfunctional blocks, steps, operations, methods, etc. using orderingsdifferent than those described herein.

References herein to “one embodiment,” “an embodiment,” “an exampleembodiment,” or similar phrases, indicate that the embodiment describedcan include a particular feature, structure, or characteristic, butevery embodiment can not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it would be within the knowledge of persons skilled in therelevant art(s) to incorporate such feature, structure, orcharacteristic into other embodiments whether or not explicitlymentioned or described herein. Additionally, some embodiments can bedescribed using the expression “coupled” and “connected” along withtheir derivatives. These terms are not necessarily intended as synonymsfor each other. For example, some embodiments can be described using theterms “connected” and/or “coupled” to indicate that two or more elementsare in direct physical or electrical contact with each other. The term“coupled,” however, can also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other.

The breadth and scope of this disclosure should not be limited by any ofthe above-described exemplary embodiments, but should be defined only inaccordance with the following claims and their equivalents.

What is claimed is:
 1. A computer implemented method, comprising:executing, by at least one processor, a collection routine at regularintervals to gather raw data from a plurality of sensors; receiving, bythe at least one processor, results of the collection routine in a formof the raw data from the sensors; storing, by the at least oneprocessor, the received results in a central data repository; setting,by the at least one processor, a device remote in a low-power mode;analyzing, by the at least one processor, the stored sensor data todetermine when a television is turned on or off, and when it is turnedon, using, by the at least one processor, a WiFi radio of the televisionto initially scan for a potential presence of one or more humans withina predetermined geographical range; and when one or more humans aredetermined by the at least one processor to be potentially present,executing, by the at least one processor, at least one further action;when the television is turned off, executing, by the at least oneprocessor, at least one further action.
 2. The method of claim 1,wherein the executing of at least one further action when the televisionis turned on comprises: fully powering up the device remote by the atleast one processor so that the device remote transitions from alow-power mode to a full power mode; performing, by the at least oneprocessor, a full-presence detection using a WiFi radio of the deviceremote to scan at a full transmission power for the presence of one ormore humans within the predetermined geographical range; when one ormore humans are determined to be present through said analyzing,determining, by the at least one processor, a respective geographicposition of the one or more humans.
 3. The method of claim 1, whereinthe sensors form a high-resolution detection zone in front of the atleast one processor, in 2 or 3 dimensions in a shape configurable byplacement of the sensors.
 4. The method of claim 1, wherein theexecuting of at least one further action when the television is turnedoff further comprises: periodically sending, by the at least oneprocessor, a low power WiFi beacon from the television to the deviceremote; entering, by the at least one processor, a low power detectionmode on the device remote; using, by the at least one processor, a WiFiradio of the device remote to initially scan at a low transmission powerfor the potential presence of one or more humans within thepredetermined geographical range; when one or more humans are determinedby the at least one processor to be potentially present, fully poweringup the device remote by the at least one processor so that ittransitions from a low-power mode to a full power mode; performing, bythe at least one processor, a full-presence detection using a WiFi radioof the device remote to scan at a full transmission power for thepresence of one or more humans within the predetermined geographicalrange; when one or more humans are determined to be present through saidanalyzing, determining, by the at least one processor, a respectivegeographic position of the one or more humans.
 5. The method of claim 2,wherein the full-presence detection using the WiFi radio of the deviceremote to scan at a full transmission power for the presence of one ormore humans within the predetermined geographical range furtherincludes: transmitting signals to the television and an internet routerusing the WiFi radio at full power, by the at least one processor;gathering, by the at least one processor, raw data from the sensors ofthe device remote; storing, by the at least one processor, the receivedraw data from the sensors of the device remote in the central datarepository; and analyzing, by the at least one processor, the storedremote sensor data to determine if one or more humans are presentproximate to the at least one processor within the predeterminedgeographical range.
 6. The method of claim 5, wherein the analyzing ofthe stored remote sensor data to determine if one or more humans arepresent proximate to the at least one processor within the predeterminedgeographical range further comprises: feeding, by the at least oneprocessor, the raw data as input to a neural network machine learningclassifier, the neural network machine learning classifier having aninput layer that receives the raw data as a plurality of inputs, and anoutput layer; and comparing, by the at least one processor, values ofnodes in the output layer to determine a presence or absence of one ormore humans proximate to the at least one processor in the predeterminedgeographical range.
 7. The method of claim 6, wherein the analyzing ofthe stored remote sensor data to determine if one or more humans areproximate to the at least one processor within the predeterminedgeographical range further comprises: comparing, by the at least oneprocessor, the determined presence or absence of humans by neuralnetwork machine learning to a known presence or absence of a userthrough user feedback; and conducting, on the basis of the comparisonthrough the user feedback, backpropagation of the neural network machinelearning classifier by the at least one processor to adjust weights ofnodes of the input layer.
 8. A system, comprising: a memory; and atleast one processor coupled to the memory and configured to: execute acollection routine at regular intervals to gather raw data from aplurality of sensors; receive results of the collection routine in aform of the raw data from the sensors; store the received results in acentral data repository in the memory; set a device remote in alow-power mode; analyze the stored sensor data to determine when atelevision is turned on or off, and when it is turned on, use a WiFiradio of the television to initially scan for the potential presence ofone or more humans within a predetermined geographical range; and whenone or more humans are determined to be potentially present, execute atleast one further action: when the television is turned off, execute atleast one further action.
 9. The system of claim 8, wherein to executethe at least one further action when the television is turned on, the atleast one processor is further configured to: fully power up the deviceremote such that the device remote transitions from a low-power mode toa full power mode; perform a full-presence detection using a WiFi radioof the device remote to scan at a full transmission power for thepresence of one or more humans within the predetermined geographicalrange; when one or more humans are determined to be present through saidanalyzing, determine a respective geographic position of the one or morehumans.
 10. The system of claim 8, wherein the sensors form ahigh-resolution detection zone in front of the at least one processor,in 2 or 3 dimensions in a shape configurable by placement of thesensors.
 11. The system of claim 8, wherein the executing of at leastone further action when the television is turned off further comprises:periodically send a low power WiFi beacon from the television to thedevice remote; enter a low power detection mode on the device remote;use a WiFi radio of the device remote to initially scan at a lowtransmission power for the potential presence of one or more humanswithin the predetermined geographical range; when one or more humans aredetermined by the at least one processor to be potentially present,fully power up the device remote so that it transitions from a low-powermode to a full power mode; perform a full-presence detection using aWiFi radio of the device remote to scan at a full transmission power forthe presence of one or more humans within the predetermined geographicalrange; when one or more humans are determined to be present through saidanalyzing, determine a respective geographic position of the one or morehumans;
 12. The system of claim 9, wherein the full-presence detectionusing the WiFi radio of the device remote to scan at a full transmissionpower for the presence of one or more humans within the predeterminedgeographical range further comprises: transmit signals to the televisionand an internet router using the WiFi radio at full power; gather rawdata from the sensors of the device remote; store the received raw datafrom the sensors of the device remote in the central data repository;and analyze the stored remote sensor data to determine if one or morehumans are present proximate to the at least one processor within thepredetermined geographical range.
 13. The system of claim 12, whereinthe analyzing of the stored remote sensor data to determine if one ormore humans are proximate to the at least one processor within thepredetermined geographical range further comprises: feed the raw data asinput to a neural network machine learning classifier, the neuralnetwork machine learning classifier having an input layer that receivesthe raw data as a plurality of inputs, and an output layer; and comparevalues of nodes in the output layer to determine a presence or absenceof one or more humans proximate to the at least one processor in thepredetermined geographical range.
 14. The system of claim 13, whereinthe analyzing of the stored remote sensor data to determine if one ormore humans are proximate to the at least one processor within thepredetermined geographical range further comprises: compare thedetermined presence or absence of humans by neural network machinelearning to a known presence or absence of a user through user feedback;and conduct, on the basis of the comparison through user feedback,backpropagation of the neural network machine learning classifier toadjusts weights of nodes of the input layer.
 15. A non-transitorycomputer-readable device having instructions stored thereon that, whenexecuted by at least one computing device, cause the at least onecomputing device to perform operations comprising: executing acollection routine at regular intervals to gather raw data from aplurality of sensors; receiving results of the collection routine in aform of the raw data from the sensors; storing the received results in acentral data repository; setting a device remote in a low-power mode;analyzing the stored sensor data to determine when a television isturned on or off, and when it is turned on, using a WiFi radio of thetelevision to initially scan for the potential presence of one or morehumans within a predetermined geographical range; and when one or morehumans are determined to be potentially present, executing at least onefurther action; when the television is turned off, executing at leastone further action.
 16. The device of claim 15, wherein the operation ofexecuting at least one further action when the television is turned oncomprises: fully powering up the device remote so that the device remotetransitions from a low-power mode to a full power mode; performing afull-presence detection using a WiFi radio of the device remote to scanat a full transmission power for the presence of one or more humanswithin the predetermined geographical range; when one or more humans aredetermined to be present through said analyzing, determining arespective geographic position of the one or more humans.
 17. The deviceof claim 15, wherein the operation of executing of at least one furtheraction when the television is turned off further comprises: periodicallysending a low power WiFi beacon from the television to the deviceremote; entering a low power detection mode on the device remote; usinga WiFi radio of the device remote to initially scan at a lowtransmission power for the potential presence of one or more humanswithin the predetermined geographical range; when one or more humans aredetermined to be potentially present, fully powering up the deviceremote so that it transitions from a low-power mode to a full powermode; performing a full-presence detection using a WiFi radio of thedevice remote to scan at a full transmission power for the presence ofone or more humans within the predetermined geographical range; when oneor more humans are determined to be present through said analyzing,determining a respective geographic position of the one or more humans.18. The device of claim 16, wherein the operation of full-presencedetection using the WiFi radio of the device remote to scan at a fulltransmission power for the presence of one or more humans within thepredetermined geographical range further includes: transmitting signalsto the television and an internet router using the WiFi radio at fullpower; gathering raw data from the sensors of the device remote; storingthe received raw data from the sensors of the device remote in thecentral data repository; and analyzing the stored remote sensor data todetermine if one or more humans are present proximate to the computingdevice within the predetermined geographical range.
 19. The device ofclaim 18, wherein the operation of analyzing of the stored remote sensordata to determine if one or more humans are present proximate to thecomputing device within the predetermined geographical range furthercomprises: feeding the raw data as input to a neural network machinelearning classifier, the neural network machine learning classifierhaving an input layer that receives the raw data as a plurality ofinputs, and an output layer; and comparing values of nodes in the outputlayer to determine a presence or absence of one or more humans proximateto the at least one computing device in the predetermined geographicalrange.
 20. The device of claim 19, wherein the operation of analyzing ofthe stored remote sensor data to determine if one or more humans areproximate to the at least one computing device within the predeterminedgeographical range further comprises: comparing the determined presenceor absence of humans by neural network machine learning to a knownpresence or absence of a user through user feedback; and conducting, onthe basis of the comparison through user feedback, backpropagation ofthe neural network machine learning classifier to adjusts weights ofnodes of the input layer.